1. Field of the Invention
The present invention relates to a method for separating images taken at different in-focus positions into an object and a backdrop and, in particular, to an image processing apparatus configured to separate into the object and the backdrop by considering the difference in the image magnification of the object between images taken serially.
2. Description of the Related Art
As a technique for taking a picture, there is a method of taking an image in a state of blurring a backdrop that is a part other than a focused object image by reducing the depth of field to narrow a focused image. In order to take an image where the depth of field is reduced, it is required that an aperture stop of a camera is opened to make the focus length longer. Generally, a single reflex camera that has an image pickup lens of large diameter and is capable of selecting variety of combinations of the aperture stop and a shutter speed can take an image using the above technique.
On the other hand, a compact digital still camera and a camera-equipped cell-phone capable of taking an image that are popular in late years employ an image pickup lens having a small lens aperture and a very short focal length to reduce the size, and therefore the depth of field is very deep. Accordingly, it is difficult to take an image where the backdrop is blurred while the above technique is used.
A method of generating an image having a blur emphasized more than that of an image taken by driving a real aperture or lens is proposed. Hereinafter, the method is referred to as “backdrop blur”.
In order to achieve the backdrop blurring, a processing for separating an area of a main object, such as a person, from areas of other objects, such a backdrop, is required. As one of the processing methods, two images are taken at different focus positions, and an area of a main object and an area of the backdrop are separated depending on the change of an edge amount between both of the images.
In case where a user takes images at different focus positions in a stage of having the camera in hand, a relative position shift is caused due to a camera shake or the like in each taken image. Therefore, in order to apply the above method, it is necessary to calculate the change of the edge amount after positions of the images are matched by correcting the relative positional shift.
However, when the position shift amount for the position adjustment is calculated, a relative calculation is performed between images where the level of the blur is different from each other, and therefore the accuracy of the calculation result of the position shift is reduced.
Further, as a method for achieving the backdrop blur by the position adjustment between images taken at different focus positions, for example, Japanese patent Laid-Open No. 2002-112095 proposes a method of acquiring two sampling images between main images and of calculating a position shift amount between the main images by using the sampling images. In particular, firstly, in a single shutter operation, two continuous image pickup operations are performed at different focus positions. Secondly, in a main image A where an obtained main object is focused and a main image B where a backdrop of a main object is focused, two sampling images are taken in between when the main image A is taken and when the main image B is taken. Thirdly, the position shift amount of the main image B from the main image A is calculated using the two sampling images. Further, the main image A and the main image B where the position shift is corrected based on the calculated position shift amount are synthesized, and an image having a desired level of the blur is obtained.
However, a conventional technique described in the above-mentioned Japanese patent Laid-Open 2002-112095 takes a temporal difference between the main image and the sampling image. For example, if a camera shake is caused between when the main image is taken and when the sampling image is taken, a calculated position shift amount between the sampling images is affected by the effect of the camera shake. Therefore, an error is caused when the position shift amount between the sampling images is used as a position shift amount between the main images.
Furthermore, since the position shift amount is calculated using the sampling image which has a smaller size than the main image, the accuracy of the calculated position shift amount is low. In particular, in case where the position shift amount for the main image is calculated using ⅛ sampling images as embodiment in Japanese patent Laid-Open 2002-112095 and the calculated result is converted to the position shift amount in the main image to be eventually obtained, the position shift amount can detect the position shift amount only at each eight pixels.
The case does not consider the change of the image magnification of an object depending on the change of the focus position, and cannot perform a processing of improving the accuracy of the position adjustment.